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ASSOCIATIVE NEURON IN AN ARTIFICIAL NEUTRAL NETWORK 

BACKGROUND OF THE INVENTION 

The invention relates to an associative neuron used in artificial 
neural networks. 

5 in artificial neural networks, neurons derived from the McCullogh- 

Pitts (1943) neuron, such as different versions of the perceptron (Frank 
Rosenblatt 1957), are used. Neural networks are discussed, for example, in 
the article "Artificial Neural Networks: A Tutorial" by Anil K. Jain, Jianchang 
Mao and K.M. Mohiuddin in IEEE Computer, March 1996, p. 31 to 44. 

10 In Fig. 1, signals X, to X n are inputs of an artificial neuron and Y is 

its output signal. The values of the input signals X, to X n can be continuously 
changing (analogous) or binary quantities, and the output signal Y can usually 
be given both positive and negative values. to W n are weighting 
coefficients, i.e. synaptic weights, which can also be either positive or 

15 negative. In some cases, only positive signal values and/or weighting 
coefficients are used. Synapses 11, to 11 n of the neuron weight the 
corresponding input signal by the weighting coefficients W, to W n . A summing 
circuit 12 calculates a weighted sum U. The sum U is supplied to a 
thresholding function circuit 13, whose output signal is V. The threshold 

20 function can vary, but usually a sigmoid or a piecewise linear function is used, 
whereby the output signal is given continuous values. In a conventional 
neuron, the output signal V of the thresholding function circuit 13 is 
simultaneously the output signal Y of the whole neuron. 

When neurons of this kind are used in artificial neural networks, the 

25 network must be trained, i.e. appropriate values must be found for the 
weighting coefficients W 1 to W n . Different algorithms have been developed for 
the purpose. A neural network that is capable of storing repeatedly supplied 
information by associating different signals, for example a certain input with a 
certain situation, is called an associative neural network. In associative 

30 neurons, different versions of what is known as the Hebb rule are often used. 
According to the Hebb rule, the weighting coefficient is increased always when 
the input corresponding to the weighting coefficient is active and the output of 
the neuron should be active. The changing of the w ighting co fficients 
according to the algorithms is called the training of th n ural network. 

35 From previously known artificial neurons, it is possible to assemble 

neural networks by connecting neurons in parallel to form layers and by 
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arranging the layers one after the other. Feedback can be impl m nted in the 
networks by feeding output signals back as input signals. In wide networks 
assembled from neurons, however, the meaning of individual signals and even 
groups of signals is blurred, and the network becomes more difficult to design 
5 and manage. To produce an attention effect, for example, the network 
operations would have to be strengthened in one place and weakened in 
another, but the present solutions do not provide any clear answers to where, 
when and how this should be done, and in what way. 

BRIEF DESCRIPTION OF INVENTION 

10 The object of the invention is to provide a method and equipment 

implementing the method in which the above problems of training a neural 
network can be solved. To put it more precisely, the object of the invention is 
to provide a mechanism by which useful additional information can be 
produced on the level of an individual neuron about the relations between the 

15 different input signals of the neuron. The mechanism must be flexible and 
versatile to make artificial neurons widely applicable. The mechanism must 
also be fairly simple so that the costs of manufacturing neurons can be kept 
low. 

The object of the invention is achieved by a method and equipment 

20 that are characterized by what is stated in the independent claims. The 
preferred embodiments of the invention are claimed in the dependent claims. 

The invention is based on expansion of a conventional neuron such 
that a specific expansion, i.e. nucleus, is attached to the conventional neuron, 
a specific main input signal, i.e. main signal, passing through the nucleus. The 

25 nucleus keys and adjusts the main signal by a signal obtained from the 
conventional part of the neuron, and forms between these signals logical 
operations and/or functions needed to control neural networks. The processing 
power of a single neuron is thus increased as compared with the previously 
known neurons, which process data only by means of weighting coefficients 

30 and threshold functions. On the other hand, a clear distinction between main 
signals and auxiliary signals makes neural networks easier to design, since the 
training according to the Hebb rule is then easy to implement in such a way 
that each weighting coefficient is increased always when the main signal and 
the auxiliary input signal concerned are simultan ously active. 

35 On the basis of the main signal (S) and a non-linear signal (V), the 

function (S 0 )S OR V is formed in the neuron of the inv ntion and us d to 
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generate a main output signal, and in addition, at least one of the three logical 
functions Y 0 =S AND V, N 0 =NOT S AND V, N a =S AND NOT V is formed and 
used to generate an additional output signal for the neuron. 

The neuron of the invention and the network consisting of such 
5 neurons learn quickly: even one example may suffice. The operation of the 
neuron of the invention and that of the networks consisting of such neurons 
are simple and clear. 

BRIEF DESCRIPTION OF DRAWINGS 

The invention will now be described in greater detail by means of 

10 preferred embodiments and with reference to the attached drawings, in which 

Fig. 1 is a general view of an artificial neuron, 
Fig. 2 is a general view of a neuron of the invention, 
Fig. 3 is a block diagram of the neuron of the invention, and 
Figs. 4 to 6 illustrate ways of implementing specific details of the 

1 5 neuron of the invention. 

DETAILED DESCRPTION OF INVENTION 

In Fig. 2, the neuron according to a preferred embodiment of the 
invention comprises a main signal input S, an arbitrary number of auxiliary 

signal inputs A 1t A 2 A„ f at least one controlling input C and at least one 

20 inhibiting input I, and a number of outputs. In the example of Fig. 2 the main 
output signal of the neuron is S 0 , and Y 0 , N 0 and N a (or one/some of them) are 
auxiliary output signals. The input and output signals can be, for example, 
voltage levels. 

Blocks 21,, 21 2 , .... 21 n are synapses of the neuron, in which the 
25 weighting coefficient corresponding to the auxiliary signal A„ A 2 , .... A„ 
concerned is stored. In practice, the synapses are, for example, circuit units. 
Block 12 is a summing circuit, in which the output signals At,, .... At 3 of the 

synapses 21„ 21 2 21„ are summed. Block 13 is a thresholding circuit, 

which can be implemented simply as a comparator, which supplies an active 
30 output signal only if its input signal level, i.e. the output signal level of the 
summing circuit 12, exceeds a pre-set threshold value. 

Block 22 compris s the neuron expansions of the invention. In the 
present application, the expansions are call d the nucl us of the neuron. The 
function of the nucleus is, for xampl , to key and adjust th main signal S on 
35 the basis of the output signal of the thresholding circuit 13 and to form logical 
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operations and/or functions between the signals. Particularly us ful logical 
operations are the logical OR (signal S 0 ) and the logical AND (signal Y 0 ). 
Other logical operations can also be used in the same way as AND so that the 
main signal S is inverted first (signal N 0 ) or so that the output signal V of the 

5 thresholding circuit 13 is inverted first (signal N a ). 

In a preferred embodiment of the invention, the nucleus 22 also 
comprises circuitry that deactivates the output signal S 0 when a certain period 
of time has passed from the initiation of the signal, irrespective of what 
happens in the inputs of the neuron. The circuitry can also take care that a 

10 new output pulse cannot be initiated until a certain period of recovery has 
passed. To the nucleus 22 can also be connected an inhibiting input signal I 
(Inhibit), which inhibits all outputs when activated (forces them to an inactive 
state). The control input signal C (Control) controls the synapses' learning. 

Fig. 3 is a block diagram of a neuron of the invention, the neuron 

15 here comprising three auxiliary signal inputs A, to A n and therefore three 
synapses 21, to 21 3 in addition to the main signal input. The expanded neuron 
of the invention can be implemented in various ways within the scope of the 

inventive idea disclosed above. 

Figs. 4 to 6 show an embodiment of the neuron according to the 

20 present invention in which the input and output signals are voltage signals. In 
the embodiment of Figs. 4 to 6 the signal is called 'active', if its voltage is 
positive, and 'inactive', if its voltage is substantially zero. 

Fig. 4 shows a way of implementing the synapses 21, to 21 n of the 
neuron of Fig. 3. In this solution the voltage corresponding to the weighting 

25 coefficient of the synapse is stored through a resistor 41 and a diode 42 in a 
capacitor 43 always when auxiliary signal A, and the main signal S are 
simultaneously active. (A possible association between the main signal S and 
the key signal K is described in connection with gate 632 of Fig. 6.) The 
resistor 41 and the capacitor 43 define a time constant by which the voltage of 

30 the capacitor 43 grows. The diode 42 inhibits the voltage from discharging 
through an AND gate 40. The voltage of the capacitor 43 is supplied to an 
operational amplifier 44 functioning as a voltage follower, the input impedance 
of the amplifier being very high (i.e. the discharging of the capacitor 43 caused 
by it is negligible). Th output of the synapse is signal At,, which is obtained 

35 from input signal A, by locking it at th voltag level corresponding to the 
weighting coefficient by a diode 45 and a resistor 46. A second voltage 
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follower 47 buffers the output signal. Always when input signal A, is active, 
output signal A^ is proportional to the current value of the weighting 
coefficient. 

Fig. 5 shows a way of implementing the summing block 12 of the 
5 neuron of Fig. 3. The voltages A^ to At 3 obtained from synapses 21 , to 21 3 are 
summed by a resistor network 50 to 53. (It is readily noticeable that the 
number of the inputs At, to At 3 and that of the resistors 51 to 53 are arbitrary.) 
The thresholding is performed by a comparator 54, and the thresholding is 
here abrupt so that the output of the comparator 54 is active only when the 

10 summed voltage U in the positive input of the comparator 54 exceeds the 
threshold value in the negative input (the threshold value in the example of 
Fig. 5 being the output voltage of a constant voltage power source 55). 

Fig. 6 shows a way of implementing the nucleus 22 of the neuron of 
Fig. 3. An OR circuit 602 generates a main output signal S 0 if the inputted 

15 main signal S is active or the thresholded summed voltage V is active. The 
nucleus 22 contains a block 606, indicated by a dotted line, functioning as a 
delay circuit. In the example of Fig. 6 the delay circuit 606 comprises a buffer 
608 and an inverter 610, resistors 612 to 614 and capacitors 616 to 618. 
Normally the output of the delay circuit 606 is active, so an AND gate 604 

20 allows an output signal to pass through. When the delay caused by the 
structure of the components of the delay circuit 606 has passed, the output 
pulse, inverted, reaches the AND gate 606 and deactivates the main output 
S 0 . S 0 cannot be re-activated until the delayed output pulse in the output of 
the delay circuit 606 has ended. A logical AND operation Y 0 is formed by AND 

25 circuit 620: the first element in the operation is the main signal S and the 
second element is a summed signal V weighted by the weighting coefficients 
of the auxiliary signals A, to A n and subsequently thresholded. A 
corresponding AND operation N 0 is formed by AND circuit 622, with the 
exception that the inverse value of the main signal S has been first formed (i.e. 

30 the signal has been inverted) by NO circuit 626. The corresponding AND 
operation N a is formed by AND circuit 624, with the exception that the 
thresholded summed signal V has been first inverted by NO circuit 628. All the 
outputs can be inhibited by an I signal, which is (here) inverted by NO circuit 
630 and then supplied, in the inverted form, to AND circuits 620 to 624. The 

35 synapses are controlled by a K signal in accordanc with the Hebb rul (cf. 
Fig. 2). A control signal C is used to define when learning is allowed at all. The 
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generation of the key signal K is inhibited by AND circuit 632 when the control 

signal C is inactive. 

The additional output signals Y 0 , N 0 and N a of the neuron according 
to the invention can be used, for example, as follows. An active signal Y 0 (Y = 

5 "Yes") means that the main signal S and the auxiliary signals A correspond to 
each other, i.e. they have been associated. An active signal N 0 (N = "No) 
means that the main signal S and the auxiliary signals A do not correspond to 
each other. The auxiliary signal A is thus active, but the main signal S is not. 
An active signal N a ("No association) indicates a situation where the main 

10 signal S is active but the auxiliary signal A is not. One characteristic of the 
neural network is its ability to predict a situation. An active signal N a indicates 
that there is a new input signal S which is not predicted by the auxiliary signals 
A. Signal N a is thus a 'surprise indicator", which can be used to draw attention 

to new, surprising signals. 

15 The control signal C controls, or keys, the K signal. It is not 

expedient for the network to learn all the situations that occur. When a normal 
human being encounters a new situation, he/she either concludes or 
instinctively knows whether the situation is worth learning. This kind of 
focusing of attention can be simulated by the control signal C. 

20 In the above example the auxiliary signals A n to A,, can be given 

continuously changing values and the main signal S can be given two different 
values. The threshold function is here a simple comparative operation. The 
invention is not limited to the above, but it can be applied more broadly, for 
example, so that the main signal S and the key signal K can also be given 

25 continuous values. The threshold function can be replaced with any 
appropriate non-linear continuous or step function. The neuron's learning is 
then not limited to two mutually exclusive situations: allowed or inhibited. 
Instead, the learning process is divided into different degrees or it is a 
continuum of degrees, whereby the strength of the K signal is adjusted on the 

30 basis of the main signal S. In the normal state of the neural network (when the 
network is not being trained), the key signal K is not more than a fraction of the 
main signal S, if the S signal is active. When the network is to be trained, the 
value of the key signal K approaches the value of the main signal S. In 
practice, the binary AND gates in Figs. 4 and 6 should be replaced, for 

35 example, with analogue multipliers or adjustable amplifiers or attenuators or 
the like. 
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In practice, a huge number of neurons (usually 10 4 to 10 6 ) are 
needed in neural networks. The neuron of the invention can be implemented 
by a process suitable to large-scale integration, for example by the EEPROM 
technique, which is used to manufacture the speech storage circuits 
5 implemented by semi-conductors. Alternatively, the neurons and the neural 
network can be simulated by a computer program executed in a digital 
processor. The values corresponding to the weighting coefficients of the 
synapses of the neurons are here stored in memory locations (e.g. in a matrix 
variable) and the other parts of the neuron are implemented by software logic. 

10 The invention can be applied in areas where information is 

processed using extensive artificial neural networks. The areas include, for 
example, processing of audiovisual information, interpretation of sensory 
information in general and of speech and image in particular, and formation of 
response. The invention is applicable in many modern fields of industry, such 

15 as human/machine interfaces, personal electronic assistants and/or means of 
communication, multimedia, virtual reality, robotics, artificial intelligence and 
artificial creativity. 

It will be obvious to a person skilled in the art that with the 
advancement of technology, the basic idea of the invention can be 

20 implemented in many different ways. The invention and its embodiments are 
thus not limited to the above examples but they can vary within the scope of 
the claims. 
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CLAIMS 

1 . A method of forming output signals of an associative artificial 
neural network by 

receiving a number of auxiliary signals (A, to AJ; 
5 forming a corresponding weighting coefficient (W 1 to W n ) for each 

auxiliary signal (A, to AJ; 

forming from the auxiliary signals (A 1 to A n ) a sum (U) weighted by 
the corresponding coefficients (W, to W n ); 

applying a non-linear function to the weighted sum (U) to generate 
10 a non-linear signal (V); 

characterized by 

receiving a main signal (S), which can be associated with the 
auxiliary signals (A, to A n ) such that the weighting coefficient QJ\I : to W n ) of 
each auxiliary signal is increased when the main signal (S) and the 
15 corresponding auxiliary signal (A, to A n ) are simultaneously active; 

forming, on the basis of the main signal (S) and the non-linear 
signal (V), a function (S 0 ) S OR V t which is used to generate a main output 
signal, and at least one of the three logical functions Y 0 =S AND V, N 0 =NOT S 
AND V, N a =S AND NOT V ( and using said logical function to generate an 
20 additional output signal for the neuron. 

2. An associative artificial neuron comprising 

means (1^ to 11 n ; 2^ to 21 n ) for receiving a number of auxiliary 
signals (A 1 to An) and forming a corresponding coefficient (W 1 to W n ) for each 
auxiliary signal (A, to AJ; 
25 means (12) for forming from the auxiliary signals (A, to AJ a sum 

(U) weighted by the corresponding coefficients (W, to W n ); 

means (13) for applying a non-linear function to the weighted sum 
(U) to generate a non-linear signal (V); 

characterized by further comprising 
30 means for receiving a main signal (S) ), which can be associated 

with the auxiliary signals (A, to An) such that the weighting coefficient Ofy to 
W n ) of each auxiliary signal is increased when the main signal (S) and the 
corresponding auxiliary signal (A 1 to An) are simultaneously active; and 

means (22) for forming, on the basis of th main signal (S) and the 
35 non-linear signal (V) f a function (S 0 ) S OR V, which is us d to generate a main 
output signal, and at least one of the three logical functions Y 0 =S AND V, 
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N 0 =NOT S AND V, N a =S AND NOT V, and using the thus obtained logical 
function to generate an additional output signal for the neuron. 

3. A neuron according to claim 2, characterized by said 
non-linear function being a threshold function and the non-linear signal (V) 

5 obtained by the function having a first state and a second state. 

4. A neuron according to claim 2, characterized by the non- 
linear function being a step function with more than two steps. 

5. A neuron according to claim 2, characterized by the non- 
linear function being a continuous function. 

10 6. A neuron according to claim 2, c h a r a c t e r i z e d by the main 

output signal (S 0 ) having a first state and a second state and the neuron 
further comprising means (606) for setting an upper limit to the length of time 
that said main output signal (S 0 ) is in the second state. 

7. A neuron according to claim 6, characterized by the 
15 neuron further comprising means (606) for setting a lower limit to the length of 

time that said main output signal (S 0 ) remains in the first state after having 

been in the second state. 

8. A neuron according to any one of claims 2 to 7, character- 
ized by further comprising means (40, 632) for adjusting the neuron's 

20 learning in response to an external control signal (C). 

9. A neuron according to claim 8, characterized by the 
means (40, 632) for adjusting the neuron's learning having two states, 
whereby the neuron's learning is either allowed or inhibited. 

10. A neuron according to any one of claims 2 to 9, charac- 
25 t e r i z e d by further comprising means (630, 604, 620 to 624) for forcing at 

least one ouput signal (S 0 , Y 0 , N Q , N a ) to a predetermined state in response to 
an external inhibiting signal (I). 
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